User interface device provided with surface haptic sensations

ABSTRACT

A user interface device includes a housing, a touch sensitive surface supported by the housing, a haptic output device supported by the housing, and a processor disposed within the housing. The processor is configured to receive a gestural input from the touch sensitive surface, generate a signal for a haptic effect to be generated by the haptic output device, and output the signal to the haptic output device to generate the haptic effect.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application claiming the benefit ofpriority from U.S. patent application Ser. No. 13/840,876, filed Mar.15, 2013, the entire content of which is incorporated herein byreference.

FIELD

The present invention is related to a user interface device providedwith surface haptic sensations.

BACKGROUND

Current user input elements of user interface devices, such as gamingperipherals, typically have a fixed static feel according to theirmaterial properties. For example, gaming peripherals, such as gamepads,are used for a wide range of games and are used for control in a widerange of gaming situations. Despite the great variety of situations inwhich a gamepad is used in a game, the traditional user input elements,such as joysticks, buttons, d-pads, and triggers, all have a fixedmechanical surface feel that is dependent on the materials that are usedto manufacture them. In addition, standard gaming peripherals currentlyonly use whole body vibration to provide tactile feedback to the user.

SUMMARY

It is desirable to have the ability to change the feel of the surfacefeatures on surfaces of user interface devices, such as the surfaces ofuser input elements of gaming peripherals, to provide a whole newexperience for game players. For example, it is desirable to have theability to change the surface properties of the device according to thefunction the user is controlling, or the game situation the user isexperiencing, to enhance the immersion feeling in the game. By providingthe surfaces of user input elements on gaming peripherals to have, forexample, a programmable friction sensation, the user input elements maycommunicate additional information to the user, such as game state. Thismay also allow a controller of a gaming system to adapt to the gamingscenario currently engaged in to make the gaming peripheral, such as agamepad or wand, feel more like the virtual object the user isexperiencing.

According to an aspect of the invention, there is provided a userinterface device that includes a housing, a user input element supportedby the housing, and a haptic output device supported by the housing. Thehaptic output device is configured to generate a haptic effect at asurface of the user interface device. The surface is part of the housingand/or the user input element. The user interface device includes aprocessor disposed within the housing. The processor is configured toreceive an input command from the user input element, communicate theinput command to a host computer, receive an output command from thehost computer, and output a haptic signal based on the output command tothe haptic output device to generate the haptic effect at the surface.

According to an aspect of the invention, there is provided a system thatincludes a display configured to display video content, a host computercomprising a first processor configured to output the video content tothe display, and a user interface device in communication with the hostcomputer. The user interface device includes a housing, a user inputelement supported by the housing, and a haptic output device supportedby the housing. The haptic output device is configured to generate ahaptic effect at a surface of the user interface device. The surface ispart of the housing and/or the user input element. The user interfacedevice includes a second processor disposed within the housing. Theprocessor is configured to receive an input command from the user inputelement, communicate the input command to the host computer, receive anoutput command from the host computer, and output a haptic signal basedon the output command to the haptic output device to generate the hapticeffect at the surface.

BRIEF DESCRIPTION OF THE DRAWINGS

The components of the following Figures are illustrated to emphasize thegeneral principles of the present disclosure and are not necessarilydrawn to scale. Reference characters designating correspondingcomponents are repeated as necessary throughout the Figures for the sakeof consistency and clarity.

FIG. 1 schematically illustrates a system in accordance with embodimentsof the invention;

FIG. 2 schematically illustrates a user interface device of the systemof FIG. 1 in accordance with embodiments of the invention; and

FIG. 3 schematically illustrates a processor of the system of FIG. 1 inaccordance with embodiments of the invention.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates an embodiment of a system 100 inaccordance with embodiments of the invention. As illustrated, the system100 includes a user interface device 110, a host computer 140, and adisplay 160. In an embodiment, the system 100 is a gaming system, theuser input device 110 is a gaming peripheral or controller, and the hostcomputer 140 is a gaming console, and as discussed in further detailbelow.

The user interface device 110, which is illustrated in further detail inFIG. 2, includes a housing 112, user input elements 114 supported by thehousing 112, a haptic output device 116 (FIG. 2) supported by thehousing 112, and a processor 118 (FIG. 2) disposed within the housing112. In an embodiment, the user interface device 110 may be a computermouse. In an embodiment, the user interface device 110 may be a gamingperipheral, such as a gamepad, a wand, a remote control, a forcesensitive plate, a steering wheel, a computer mouse, or any other devicethat is configured to allow a user to provide input to the host computer140 and receive haptic effects from the haptic output device 116, asdiscussed in further detail below. For example, the user interface 110may be a wearable device, such as a pair of glasses configured to allowa user to provide input to the host computer 140 and/or receive hapticeffects from the haptic output device 116.

The housing 112 may be formed from one or more materials that providedifferent functionalities to the user interface device 110. For example,a portion of the housing 112 may include a stiff polymeric material thatis suitable for supporting the user input elements 114, the hapticoutput device 116, and the processor 118. In an embodiment, a portion ofthe housing 112 may include a conductive material so that the hapticoutput device 116 may provide a particular type of haptic effect to theuser, as discussed in further detail below.

The user input elements 114 may include one or more of a button 114 a, ad-pad 114 b, an analog joystick 114 c, a trigger 114 d, a touchscreen114 e, and a touch sensitive surface 114 f, as illustrated in FIG. 2.Each of the user input elements 114 is configured to allow the user toprovide an input command to the host computer 140 via the processor 118,as discussed in further detail below.

The haptic output device 116 is configured to generate a haptic effector sensation to the user of the user interface device 110. The hapticeffects or sensations that are generated by the haptic output device 116may be created with any method for creating haptics, such as vibration,deformation, kinesthetic sensations, electrostatic or ultrasonicfriction, etc. In an embodiment, the haptic output device 116 mayinclude an actuator, for example, an electromagnetic actuator such as anEccentric Rotating Mass (“ERM”) in which an eccentric mass is moved by amotor, a Linear Resonant Actuator (“LRA”) in which a mass attached to aspring is driven back and forth, or a “smart material” such aspiezoelectric materials, electro-active polymers or shape memory alloys,a macro fiber composite actuator, an electro-static actuator, anelectro-tactile actuator, and/or another type of actuator that providesa physical feedback such as a haptic (e.g., vibrotactile) feedback. Thehaptic output device 116 may include non-mechanical or non-vibratorydevices such as those that use electrostatic friction (ESF), ultrasonicsurface friction (USF), or those that induce acoustic radiation pressurewith an ultrasonic haptic transducer, or those that use a hapticsubstrate and a flexible or deformable surface, or those that provideprojected haptic output such as a puff of air using an air jet, orthermal devices that provide temperature effects, and so on. In anembodiment, multiple haptic output devices may be used in the userinterface device 110 to generate different haptic effects.

Returning to FIG. 1, the host computer 140 includes at least oneprocessor 150 configured to execute one or more computer programmodules, as described below with respect to FIG. 3. The host computer140 may be any type of computing device, such as a personal computer oras mentioned above, a gaming console when the system 100 is a gamingsystem, or a handheld device, such as a touch screen handheld device,which may be a mobile device. As illustrated in FIG. 1, the hostcomputer 140 is in communication with the user interface device 110though a wired or wireless communication channel 170 established betweena communication port 141 of the host computer 140 and a communicationport 111 of the user interface device 110.

As illustrated in FIG. 3, the processor 150 is configured to execute oneor more computer program modules. The one or more computer programmodules may include one or more of a content provision module 152, ahaptic effect determination module 154, a haptic output device controlmodule 156, and/or other modules. The processor 150 may be configured toexecute the modules 152, 154, and/or 156 by software, hardware,firmware, some combination of software, hardware, and/or firmware,and/or other mechanisms for configuring processing capabilities onprocessor 150.

It should be appreciated that although modules 152, 154, and 156 areillustrated in FIG. 3 as being co-located within a single processingunit, in embodiments in which the host computer 150 includes multipleprocessors, one or more of modules 152, 154, and/or 156 may be locatedremotely from the other modules. For example, as discussed in furtherdetail below, the processor 118 of the user interface device 110 mayinclude one or more of the modules 152, 154, and/or 156. The illustratedembodiment is not intended to be limiting in any way.

The description of the functionality provided by the different modules152, 154, and/or 156 described below is for illustrative purposes, andis not intended to be limiting, as any of the modules 152, 154, and/or156 may provide more or less functionality than is described. Forexample, one or more of the modules 152, 154, and/or 156 may beeliminated, and some or all of its functionality may be provided byother ones of the modules 152, 154, and/or 156. As another example, theprocessor 150 may be configured to execute one or more additionalmodules that may perform some or all of the functionality attributedbelow to one of the modules 152, 154, and/or 156.

The content provision module 152 is configured to control the provisionof content to the user of the system 100 via the display 160. If thecontent includes computer generated images (e.g., in a videogame,virtual world, augmented reality virtual world, simulation, etc.), thecontent provision module 152 is configured to generate the images and/orviews for display to the user through the display 160. If the contentincludes video and/or still images, the content provision module 152 isconfigured to access the video and/or still images and to generate viewsof the video and/or still images for display on the display 160. If thecontent includes audio content, the content provision module 152 isconfigured to generate the electronic signals that will drive a speaker,which may be part of the display 160, to output corresponding sounds.The content, or information from which the content is derived, may beobtained by the content provision module 152 from an electronic storage158, which may be part of the processor 150, as illustrated in FIG. 3,or may be separate from the processor 150.

The haptic effect determination module 154 is configured to determinethe haptic effect or sensation to be generated by the haptic outputdevice 116 of the user interface device 110 for the user, based oninformation received from the content provision module 152 or the userinput element 114 or any other input that is configured to determine theuser's intent. Determining the haptic effect may include determining oneor more parameters that include an amplitude, frequency, duration, etc.,of the haptic sensation. The haptic effect is determined by the hapticeffect determination module 156 to enhance one or more aspects of theexperience provided by the content to the user. For example, the hapticeffect may be determined to enhance one or more of the realism of thecontent, the enjoyability of content, perception of the content by theuser, and/or other aspects of the experience provided by the contentbeing conveyed to the user via the display 160.

The haptic output device control module 158 is configured to control thehaptic output device 116 to generate the haptic effect determined byhaptic effect determination module 156. This includes communicating thehaptic output signal to be generated by the processor 150 to the hapticoutput device 116. The haptic effect to be generated may be communicatedover wired communication links, wireless communication links, and/orother communication links between the processor 150 and the hapticoutput device 116, such as through the wired or wireless communicationchannel 170 established between a communication port 141 of the hostcomputer 140 and a communication port 111 of the user interface device110. In an embodiment, at least a portion of the functionalityattributed to the haptic output device control module 158 may bedisposed in the processor 118 carried by the user interface device 110and data from the haptic effect determination module 156 located in theprocessor 150 may be communicated to the haptic output device controlmodule 158 in the processor 118 through the wired or wirelesscommunication channel 170.

The haptic output device 116 is configured to receive the haptic signalfrom the haptic output device control module 158, and output a hapticeffect to the user through the user interface device 110. The hapticsensations the user feels may depend on the information beingcommunicated to the user by the host computer 140. For example, thehaptic sensation may be used to convey game information, such as objectinteraction, character movement, character status, and/or charactertype. The haptic sensations can also be used to indicate the currentmode of interaction, such as moving a character, or holding a specificweapon. The haptic sensations may also be queried in the sense that theuser may explore a surface of the housing 112 and/or user input elements114 to determine or check for specific information.

The surface of the housing 112 and/or the user input elements 114 suchas the buttons 114 a, d-pad 114 b, and joystick 114 c of the userinterface device 110 may have programmable friction, texture, such as afeel of grooves and bumps, or other haptic sensations. The hapticsensations may be created by electrostatic friction, ultrasonicfriction, local surface deformation, electrotactile stimulation, such aselectrocutaneous stimulation, thermal effects and/or vibration.

In an embodiment, surface haptic sensations may be in the form ofprogrammable textures or varying friction sensations. For example, thesurface of the d-pad 114 b may have electrodes 120 embedded in one ormore of the four sections of the d-pad 114 b. When the user moveshis/her finger or thumb over the surface of the d-pad 114 b, theelectrodes 120 may be charged in a time varying fashion in order tocreate an electrostatic force on the user's finger to create thesensation of detents, textures or a programmable friction sensation.D-pads are often used to enter directional information into the game.The user typically slides his/her thumb along this type of user inputelement 114, which is particularly suited to provide electrostatichaptic sensations.

According to embodiments of the invention, surface haptic sensations maybe created on at least one of the following user input element 114surfaces: buttons 114 a, d-pad 114 b, analog joystick 114 c, triggers114 d, touchscreen 114 e, and touch sensitive surfaces 114 f, and/or onat least one of the surfaces, i.e. front, back, or side surfaces, of thehousing 112.

In an embodiment, the friction of the user input elements 114 may beprogrammably adjusted. For example, buttons 114 a that are active in agaming application may have one friction sensation and those that arenot used in the gaming application may have another level of frictionsensation, so the user may feel the active button through hapticfeedback.

In an embodiment, a user may move his/her finger along the back of thehousing 112 in order to query for information, such as boss strength orcharacter status.

In an embodiment, programmable surface detents may be placed on thed-pad 114 b when the user's character is interacting with differentsurfaces in the game. For example, a different sensation may be feltwhen the user's character is running through mud as compared to whenrunning on hard ground.

In an embodiment, the top of the analog joystick 114 c may have aprogrammable texture to indicate game environment or interaction mode.For example, a different texture may display when a user is driving adifferent vehicle or is using a different weapon.

In an embodiment, the triggers 114 d may have a different texture orfriction depending on weapon type, firing rate, etc. In an embodiment,the user may set the feel of the haptic effect based on the user'spreference.

In an embodiment, the user interface device 110 may be a wand orwand-like motion controller that has programmable textures or frictionon the housing 112 to indicate weapon type or specific ways objects arebeing interacting with. For example, an ice sword may have a lowsimulated friction coefficient compared to a regular sword.

In an embodiment, one of the user input elements 114 described above,such as one of the buttons 114 a, or another user input element on theuser interface device 110 may be used to adjust the level of frictionon, for example, the d-pad 114 b.

In an embodiment, electronic circuitry may be included in the userinterface device 110 to drive the haptic output device 116 based on theoutput of the haptic output device control module. For example, anamplifier may be used to reach the voltages need for certain types ofactuators to generate the desired haptic effect.

According to embodiments of the invention, the haptic output device 116is coupled directly to, or integrated with, the surface to be augmentedwith surface haptics, depending on the desired haptic effect.

In embodiments in which at least part of the haptic effect includeselectrostatic friction, the outer surface of the housing 112 or thesurface of the input element 114 providing the surface of the userinterface device 110 to be augmented is made of a conductive materialcovered with a thin insulator. For example, one of the buttons 114 a maybe made of metal and covered with a thin layer of plastic. Theconductive layer is connected to the driving electrical signal providedby the haptic output device 116. In this embodiment, the user's bodymust be connected to a shared ground, either through another surface ofthe device (e.g., back of the housing 112) or by some other method, suchas a bracelet worn by the user or a grounded chair on which the user issitting. In an embodiment, the roles of the ground and surface may bereversed, as is done with reverse electrovibration.

In an embodiment in which at least part of the haptic effect includesultrasonic friction, the surface to be augmented vibrates at ultrasonicfrequencies. Such an effect may be done using actuators, such aspiezoelectric actuators. In this embodiment, the surface is typicallysuspended at vibration nodes so that vibrations are not dampened. Forexample, a piezoelectric actuator may be attached to a d-pad that issuspended on springs.

In an embodiment in which at least part of the haptic effect includeslocal surface deformation, deformable actuators, such as macro fibercomposite (MFC) actuators may be used at the surface to be actuated toeither deform directly or move an intermediate object to deform thesurface. For example, a deformable actuator may bend at the surface of ad-pad to create a concave shape. Surface feel may also be adjusted byhaving small protrusions at the surface.

In an embodiment in which the haptic effect is electrotactilestimulation, electrodes may be placed at the surface that is to producehaptic feedback. For example, the housing 112 may be covered with a filmcontaining an array of electrodes, or select surfaces of the user inputelement 114, such as the d-pad 114 b, may be covered with a filmcontaining an array of electrodes 120, as described above. In anembodiment, the electrodes may alternatively be embedded in the materialof the surface.

In an embodiment in which the haptic effect is a thermal effect, athermal device, such as a Peltier cooler, may be used at the surface tocontrol its thermal properties, either directly or through anintermediary surface with appropriate thermal conductance.

In an embodiment in which the haptic effect is vibration, similar toultrasonic friction described above, vibrations are applied to thesurface through an actuator connected to the surface. The actuator maybe an eccentric rotating mass (ERM) actuator, a linear resonant actuator(LRA), a piezoelectric actuator, or any other vibration actuator.

In an embodiment, the user interface device 110 may include a touchsensor 122 at the surface to be augmented that is configured to sense aposition of contact with the surface by the user and communicate theposition to the processor 118 so that the haptic effect may bemodulated. For example, electrostatic friction is typically applied toan entire surface but can be modulated as a finger moves against it tocreate the illusion of local features. Such a haptic effect requiressensing of the finger position, and communicating the sensor readings tothe haptic output device control module 156 or the content provisionmodule 152 in order to adjust the haptic rendering to the surface.

Exemplary implementations of embodiments of the invention in a gamingenvironment will now be described. In an implementation, the texture ofone of the buttons 114 a may be repetitively felt by the user whilewaiting for the right timing for an action to take during a game. Forexample, a gamer could be waiting for a laser gun to recharge or for acue from the game for the performance of a timing-critical operation.The texture of the button 114 a may be configured to change gradually toindicate progress, abruptly to indicate a sudden change, or very brieflyto provide a cue that can easily be missed.

In an implementation, a game may vary the friction at the surface of acontrol such as the d-pad 114 b in order to disturb the player and makethe game more difficult. For example, a car that has been damaged may becontrolled with a d-pad having unpredictable friction.

Games often involve repetitive gestures such as repeatedly pressingleft-right on a d-pad or shaking a gamepad. In an implementation, thetexture of the d-pad 114 b or a surface of the housing 112 may change toprovide feedback about the performance of an action, such as progressbeing made in the breaking of a door, or some other action requiring arepetitive gesture.

Games often have a tutorial at the beginning in which the controls aregradually explained. In an implementation, surface effects may be usedto highlight the buttons 114 a and other user input elements 114 thatare being shown in the tutorial. For example, a tutorial may explainthat a particular button needs to be pressed and add a texture to thatbutton to make it easier to find. This may also be used to help playersin “easy” mode, for example, to indicate what button needs to be pressedin certain situations. For example, a haptic effect may be provided tothe user input element 114 that is predicted to be the most likelyelement to be used next.

In an implementation using the thermal device as the haptic outputdevice 116, the surface of the housing may get hot to match events in agame, such as an explosion or shooting a gun.

In an implementation, surface haptics may also be used to indicate thepresence of virtual controls such as buttons and sliders, for example onthe touch sensitive surface 114 f, and simulate physical interactionswhile the virtual controls are used. For example, a game may usedifferent virtual controls on the backside of the housing 112, dependingon the current context, such as different mini-games with distinct inputrequirements. Touch sensitive surfaces may also be used for gesturalinput with haptic textures and effects providing confirmation andadditional feedback. This may be particularly relevant if additionalsensors are used on the surface to enable spatial tactile effects. In animplementation, the touch sensitive surfaces may also be used for freespace gestural input. For example, the surface feel of the device maychange according to the position, orientation, and/or velocity of thedevice. In an implementation, surface haptics may be used to create avirtual button that is felt by the user to be raised or depressed. Thestiffness of the virtual button may be modified with surface haptics tomake the button seem easier or harder to press.

In an implementation, the haptic state of surface of the housing 112 orat least one of the user input elements 114 may communicate ambientinformation about the current state of the user interface device 110.For example, in embodiments in which the user interface device isbattery powered, different textures of at least a portion of the surfacemay indicate different battery levels in a subtle, non-obtrusive manner.

The illustrated and above-described embodiments are not considered to belimiting in any way, and embodiments of the present invention and may beused to enable haptic feedback in various electronic devices, such astouch screen handheld devices (mobile devices, PDA, and navigationsystems), automotive applications, etc.

Although many of the examples described herein relate to gaming devices,it should be understood that the present disclosure also encompassesother types of human-computer interfaces involving touch sensitivestructures. In addition, other features and advantages will be apparentto one of ordinary skill in the art upon reading and understanding thegeneral principles of the present disclosure. These other features andadvantages are intended to be included in the present disclosure aswell.

The embodiments described herein represent a number of possibleimplementations and examples and are not intended to necessarily limitthe present disclosure to any specific embodiments. Instead, variousmodifications can be made to these embodiments as would be understood byone of ordinary skill in the art. Any such modifications are intended tobe included within the spirit and scope of the present disclosure andprotected by the following claims.

What is claimed is:
 1. A user interface device comprising: a housing; atouch sensitive surface supported by the housing; a haptic output devicesupported by the housing; and a processor disposed within the housing,the processor configured to receive a gestural input from the touchsensitive surface, generate a signal for a haptic effect to be generatedby the haptic output device, and output the signal to the haptic outputdevice to generate the haptic effect.
 2. The user interface deviceaccording to claim 1, further comprising a touch sensor at the touchsensitive surface, the touch sensor configured to sense a position ofcontact with the touch sensitive surface by a user of the user interfacedevice and communicate the position to the processor.
 3. The userinterface device according to claim 2, wherein the processor is furtherconfigured to modulate the signal to adjust the haptic effect to begenerated by the haptic output device.
 4. The user interface deviceaccording to claim 1, wherein the haptic output device is configured togenerate the haptic effect at the touch sensitive surface.
 5. The userinterface device according to claim 1, further comprising acommunication port configured to establish a communication channelbetween the user interface device and a separate host computer.
 6. Theuser interface device according to claim 5, wherein the processor isfurther configured to communicate an input command to the separate hostcomputer via the communication channel based on the received gesturalinput.
 7. The user interface device according to claim 5, wherein theuser interface device is a gaming peripheral and the host computer is agaming console.
 8. A method for generating a haptic effect to a user ofa user interface device, the method comprising: receiving a gesturalinput provided to a touch sensitive surface of the user interface deviceby the user; generating a signal for a haptic effect to be generated bya haptic output device; outputting the signal to the haptic outputdevice; and generating the haptic effect with the haptic output device.9. The method according to claim 8, further comprising: sensing aposition of contact with the touch sensitive surface of the userinterface by the user with a touch sensor.
 10. The method according toclaim 9, further comprising: modulating the signal to adjust the hapticeffect based on the sensed position of contact.
 11. The method accordingto claim 8, wherein the haptic effect is generated at the touchsensitive surface.
 12. The method according to claim 8, furthercomprising: establishing a communication channel between the userinterface device and a separate host computer.
 13. The method accordingto claim 12, further comprising: communicating an input command to theseparate host computer via the communication channel based on thereceived gestural input.
 14. The method according to claim 12, whereinthe user interface device is a gaming peripheral and the separate hostcomputer is a gaming console.
 15. A system comprising: a displayconfigured to display video content; a host computer comprising a firstprocessor configured to output the video content to the display; and auser interface device separate from the host computer and configured tobe in communication with the host computer, the user interface devicecomprising a housing; a touch sensitive surface supported by thehousing; a haptic output device supported by the housing; and a secondprocessor disposed within the housing, the second processor beingconfigured to receive a gestural input from the touch sensitive surface,and output a signal to the haptic output device to generate a hapticeffect based on the received gestural input, wherein the first processorand/or the second processor is configured to generate the signal for thehaptic effect to be generated by the haptic output device.
 16. Thesystem according to claim 15, wherein the host computer comprises afirst communication port and the user interface device comprises asecond communication port configured to establish a communicationchannel with the first communication port.
 17. The system according toclaim 16, wherein the second processor is further configured tocommunicate an input command to the first processor of the host computervia the communication channel based on the received gestural input. 18.The system according to claim 15, wherein the user interface device is agaming peripheral and the host computer is a gaming console.
 19. Thesystem according to claim 15, wherein the user interface device furthercomprises a touch sensor at the touch sensitive surface, the touchsensor configured to sense a position of contact with the touchsensitive surface by a user of the system and communicate the positionto the processor.
 20. The system according to claim 19, wherein thesecond processor is further configured to modulate the signal to adjustthe haptic effect based on the position of contact.
 21. The systemaccording to claim 15, wherein the haptic output device is configured togenerate the haptic effect at the touch sensitive surface.